[SPARK-59403][CORE] Fix remote fetch of empty deserialized RDD blocks - #58697
Open
leixm wants to merge 1 commit into
Open
[SPARK-59403][CORE] Fix remote fetch of empty deserialized RDD blocks#58697leixm wants to merge 1 commit into
leixm wants to merge 1 commit into
Conversation
leixm
force-pushed
the
SPARK-59403
branch
2 times, most recently
from
September 11, 2026 03:32
18cd856 to
bb2be72
Compare
Return block status metadata for every candidate location instead of using one representative status for all replicas. A zero-byte remote response is accepted only when the fetched location itself reports a nonempty deserialized in-memory block. Empty responses from disk replicas, external shuffle services, and locations without status metadata remain fetch failures. Retain the representative block size solely for the existing remote-fetch spill decision, and associate local disk directories with their matching location. Refresh locations and their status metadata together after fetch failures; a status-less fallback remains fail-closed for empty responses. Add regression coverage for the matching and non-matching replica cases, external shuffle service responses, status refresh, and the master metadata returned for different replica storage levels. Generated-by: OpenAI Codex
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR returns status metadata for every candidate block location in
BlockLocationsAndStatus, rather than returning one representativeBlockStatusfor all locations.BlockManagernow accepts an empty remote response only when the fetchedlocation itself reports a nonempty deserialized in-memory block. Empty
responses from disk-backed replicas, external shuffle service replicas, and
locations without status metadata continue to be treated as fetch failures.
The representative block size is retained for the existing remote-fetch spill
decision. The local disk directories are associated with their corresponding
location. When block locations are refreshed after fetch failures, Spark also
refreshes the per-location status metadata; a status-less fallback remains
fail-closed for empty responses.
The PR adds regression tests for valid empty deserialized-memory responses,
another deserialized-memory replica, disk and external shuffle service
responses, missing status metadata, and refreshed locations.
Why are the changes needed?
For deserialized in-memory blocks,
memSizeis an estimated size of JVMobjects, not the size of the serialized data sent over the network. Therefore,
an empty iterator can have a positive
memSizewhile its serialized responseis empty.
Previously, every empty response for a positive representative block size was
interpreted as a failed fetch. Allowing empty responses based only on that
representative status would be unsafe because the fetched replica can have a
different storage level. This PR evaluates the status of the actual fetched
location, allowing the valid deserialized-memory case while preserving failure
detection for disk replicas, external shuffle services, and unknown locations.
Does this PR introduce any user-facing change?
Yes. Remote reads of affected empty deserialized RDD cache blocks can now
succeed instead of being incorrectly reported as unavailable. There are no
public API or configuration changes.
How was this patch tested?
The suites completed successfully with 138 and 25 tests passed, respectively.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex